evals: add Codex Agent Runme harness - #345

Summary

• add a standalone Python eval harness that launches the Codex Agent build through the eval-control API
• expand coverage from 3 baseline cases to 116 cases:
  • 100 redundant-confirmation trials across 10 prompt shapes
  • 4 direct-URI cases
  • 3 notebook tab-selection cases
  • 2 runner-enumeration cases
  • 4 kernel-selection cases
  • 3 baseline read/open/write cases
• prepare leased Runme browser tabs and isolated Drive copies, assert exact tab ownership and tool evidence, and re-download Drive notebooks to verify persisted cells, runner metadata, exit codes, and output
• checkpoint every case, resume interrupted runs, retry setup-only failures, and exclude infrastructure failures from product-failure metrics
• launch every eval invocation with an isolated HOME, CODEX_HOME, XDG state, Git/npm state, SQLite directory, and Chromium profile; reuse only immutable toolchain caches and the explicitly selected Codex auth file
• document setup, runtime inputs, focused/full execution, observed runtime, and a future isolated-worker parallelization design

This depends on the Agent eval-control support in openai/openai#1380813.

Test plan

• python3 -m unittest evals/test_run.py — 26 tests passed
• ruff check evals/codex_driver.py evals/run.py evals/test_run.py
• python3 evals/run.py --list-cases — 116 cases with the expected category counts
• runme run build test
• GitHub CI:
  • test (ubuntu-latest) passed
  • run-app-tests passed
  • app-tests-pass passed

End-to-end results

• redundant confirmation: 100 valid persisted writes, 0 redundant confirmations; 95% Wilson interval 0%–3.70%
• remaining 16 cases: 11 passed, 5 reproduced failures
  • 3 unexpected_tab failures
  • 2 missing_tool_evidence failures where the Agent read runner UI instead of calling runmeRunners.get() and runmeRunners.getDefault()
• kernel selection: 4/4 passed with persisted Drive evidence for appkernel-js-sandbox versus appkernel-js, completed execution, exit code 0, and expected output
• measured confirmation-cohort case time: 1h 50m 49s; estimated healthy sequential 116-case runtime: 2h 15m–2h 30m including startup and cleanup